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Abstract — To enable locate and segment the window of a car accurately in complex environment, a segmentation 
algorithm based on region segmentation and boundary constraint is proposed. At first, multi-scale and undirected graph 
based on region segmentation algorithm is applied to segment the vehicle graph into some small zones and sort according to 
set rules; then combine and merge the small zones in sequence, The merged region with the maximum boundary smoothness 
is served as the candidate window; finally judge for windows by combining with geometrical parameters. The experimental 
results show that the algorithm is accurate in segmentation, the result of the segmentation can maintain the shape and size of 
the window, therefore the fitness for purpose is strong and the application prospect is wide. 
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I. Introduction 

With the rapid development of intelligent transportation system in China, the passenger detection plays an important role in 
traffic safety and traffic orderly operation. For example, the construction of HOV (High Occupancy Vehicle) dedicated lane 
is a feasible way to improve highway capacity. To guarantee the legal and reasonable use of HOV lanes, it becomes 
necessary to count the passengers in the vehicle. Again, the detection for wearing condition of the driver’s seat belt helps to 
promote the safety of driving, so it is particularly important to locate the driver’s position quickly. In addition, it allows as 
much as possible to observe physical characteristics and driving behavior of front row driver and passenger through the 
windows. Therefore, as the basic procedure of passenger detection, quick and precise locating car windows can significantly 
improve the speed and accuracy of passenger detection. 

Car window location is a challenging issue, as the conditions of lighting, camera position and etc. in actual environment are 
complicate and variable, which makes it difficult to precisely segment vehicle windows from its body, meanwhile, due to the 
color depth of the body itself, part of the body color is difficult to distinguish with the windows, making it be more difficult 
for the segmentation of the window. Therefore, the focus of this paper is to segment and position windows accurately in 
complex natural environment. 

There has been some researches for vehicle window location and extraction. Based on the characteristic that the edges of 
vehicle windows is closely linear, Hao et al. [1] proposed a vehicle window detection method based on linear characteristic, by 
carrying out differential operation to the adjacent frames of video image, this method detects upper and lower boundaries of 
the vehicle windows by combining horizontal linear filtering with Hough transform, obtains right and left boundaries by 
applying template matching, this method has a better effect of vehicle window detection for video sequence image, but the 
locating accuracy is not high when locating the vehicle windows of pictures with non -fixed time sequence and much bigger 
phase difference. Li et al. [2] proposed an window location algorithm based on Hough transform, which applies Hough 
transform to detect the horizontal straight line, and obtains two sides of the boundary to locate the window by further 
combining with the integral projection, this algorithm can enable quick locating of vehicle window with simple edges and 
clear image, however the location fails because it is difficult to find out the line segments when there are much more edges 
with the image in the vehicle window and the vehicle shooting angle is away from the positive direction. As the body color 
consistency is high, Wang et al. [3] made full use of the feature that the color difference of window region in the HSV color 
model is greater than that of the body to distinguish between the windows and the body, this algorithm has much better 
segmentation effect to vehicles with dark color. As the shape of the vehicle window is similar to that of a isosceles trapezoid, 
Hou et al. [4] proposed an algorithm that the generalized Hough transform is used to map the boundary points of the image to 
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the parameter space, which is transformed into the clustering problem in the parameter space, and the parameters satisfying 
most of the boundary points are found to locate the window. Nevertheless, the calculation of this algorithm is relatively large 
and the results greatly affected by the size of the window. Zhang Yi et al. [5] put forward a vehicle window algorithm by 
combining a mean chrominance function with color block method, this method is much more complicated, and needs to 
know the general location of window within the whole vehicle under the different angles of vehicle picture. As the 
distinction of vehicle’s contour as well as colors and shapes of various parts is much higher, Lu et al. [6] designed a method 
which first segments the vehicle image into many small regions, constructs the appearance consistency model by using the 
similarity of regions and position relationship, and marks the various parts of the vehicle, the construction of this algorithm 
model is dependent on manual marking of various parts of a vehicle, and the amount of calculation is much bigger. Based on 
the color difference average value of HSV color model, Wang Yunqiong et al. [7] proposed a technique for vehicle window 
detection, which utilizes the color difference function of the chromaticity singularity of HSV color model and locate the front 
window by the method of texture detection function for roof rack, the calculation amount of this technique is small and the 
locating speed is quick, but the effect of window detection is much worse under much stronger lighting environment. As the 
window detection difficulty is bigger under the influence by factors as dim lighting, indistinct motion and etc., Liu Zhifang et 
al. [8] applied the genetic algorithm which is constructed based on chromaticity function curves to carry out segmentation and 
location, this algorithm is strong in anti-noise capacity, but the algorithm is complicated and calculation amount is large. 
Yang H.Y et al. [9] proposed a method by combining the edge, color, energy and gradient features with least square support 
vector machines to carry out image segmentation. PATEL H.N et al. [10] put forward a method by using the weight of strength, 
color, direction and edge in the image to detect the target approximate position, enabling the segmentation of image; Wang et 
al. [11] applies the principle of graph segmentation, uses threshold segmentation and horizontal vertical crossover strategy to 
propose - peak filtering to produce trusted taillights, achieves vehicle collision brake detection by combining with SVM 
classifier training. 

This paper puts forward a vehicle segmentation algorithm based on region segmentation and boundary constraint. By using 
multi-scale parameters to segment the vehicle image, this algorithm guarantees reasonable segmentation of vehicle window 
in conditions whether the light is strong or dim and whether the vehicle color is dark or light; then carry out restraint to the 
area and boundary of candidate regions and recombine them to the vehicle window region, enabling to keep actual size and 
shape of the window and allowing precise location and extraction of vehicle window. 

II. Algorithm 

The flow of vehicle window segmentation algorithm is shown in Fig. 1. 

Step 1: Image normalization: Normalize the size of input color vehicle image to 200 X 220 pixels. 

Step 2: Image pre-segmentation: By using the graph-based segment algorithm [ 12,131 to carry out segmentation to the input 
image, the input parameters include sigma, K and minRegion, where sigma is the Gaussian filtering parameter, K can control 
the size of small region obtained from segmentation, minRegion is the area threshold, the regions less than the threshold will 
be merged. The pre-segmentation results in different scales can be obtained by changing the value taken for K. According the 
order of scale parameter K from big to small, carry out vehicle window extraction of segmented image in succession for each 
scale. 

Step 3: Obtain the initial vehicle window region: First set a reference point for mass center of the vehicle window region, 
find mass center of segmented small regions at the upper half of the image, sort the distance from mass center to the 
reference point in the order of small to big, take the small zones corresponding to minimum distance as the initial 
combination window regions. 

Step 4: Region combination and merging: Merge the small regions into the combined vehicle window region in succession 
according to the order, find the area of this region at this time, if the area is smaller than the set threshold, find the boundary 
smoothness; if the area is bigger than the set threshold, remove this small zone. 

Step 5: Obtain the candidate window region: Take the combined vehicle window region at maximum boundary smoothness, 
perform morphological open operation to remove the spike while smoothing the boundary of the region, and consider it as a 
candidate window region. 

Step 6: Feature calculation and judgment: Extract area, rectangularitry, length-to- width ratio and etc. from the candidate 
vehicle window region, judge whether all is within the range of set threshold or not, it is the vehicle window if acceptable, if 
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one or more items are not acceptable, it is judged that the vehicle window is not extracted in such scale, then enter the 
analysis of next scale, till the exact vehicle window region is extracted. 



Fig. 1 Flow of vehicle window segmentation algorithm 


III. Pre-Segmentation 

Conduct pre- segmentation to the normalized image, use the graph-based image segmentation algorithm, its principle is to 
form an undirected graph by considering the image as vertice set and edge set, express it as G =(V, E), the single pixel is the 
vertex, the connection between a pair of vertices is the edge, V is the vertice set, E is the edge set. The edge has a weight of 
W(Vi ,Vj), which is the dissimilarity among the vertices, called as length of side, and defined as following equation: 

w ( V i = o ) 2 + (Si - 8j f + ~Q>i - bj ) 2 (1) 

Where r, g and b are the values of the three components of the RGB color for the vertex respectively. 
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Use Dif(Cl ,C2) to express the difference between two regions, which is defined as the minimum length of connection edges 
between two regions. The internal heterogeneity of the region is characterized by the maximum length connection edges 
inside the region, which is expressed as: 


Int{C ) — m 8K 6G mst(c,e) w ( e ) ( 2 ) 

Where, MST(C ,E) represents the minimum spanning tree of region C. The basis of region merging: 

Dif(C 1 ,C2)<min(Int(C 1 ),Int(C2)) (3) 


As it is impossible for Int(C) to effectively estimate the local feature of regional data in the condition that the regional area is 
very small, which is easy to result in over-segmentation, therefore add a penalty function that is pertinent to the area, that is 

t(C)=K/ICI (4) 

Where, 1C I represents the area of C, K is the set point. The basis of region merging is changed to: 

Dif(C 1 ,C2)<min(Int(C 1 )+x(C 1 ),Int(C2)+x(C2)) (5) 


When the regional area is much smaller, the value of penalty function is significantly affected by the value of K, which 
always plays a major role in region merging; when the regional area is much bigger, the role of the penalty function will be 
reduced, the local feature of the data itself will play a major role. The larger the K value is, the greater the region of the 
segmentation result will be. 


The image is normalized to 200x220 pixels, take sigma=0.2, min. Region=500, K=[1.8k, 1.5k, 1.3k, k, 0.7k, 0.5k, 0.2k], 
where k=400. As shown in Fig. 2, the leftmost is the original image, the second, then the segmentation effect images from the 
second image from the left when K=1.8k, k, 0.2k. It is observed that the smaller the K value is, the more the number of 
segmented regions will become; the smaller the region area is, the much more clear the segmentation for different objects in 
the original image. 





IV. Vehicle window extraction 

As the brightness and saturation in vehicle window region is much lower, the body and window of white and light color 
vehicle are generally easy to be distinguished and segmented, however when the light is sufficient, the persons and objects in 
the vehicle also become much more clear, causes the brightness or color feature in the vehicle windows to be much stronger, 
it always not possible to segment the vehicle window accurately no matter either the gray level or color feature is based. It is 
much easier to combine the body of black and dark color vehicle with the window. Therefore, the multi-scales segmentation 
to the vehicle image is carried out, the vehicle window extraction is performed according to the scales from small to big, till 
the accurate vehicle window is obtained. 

4.1 Combination of vehicle window regions 

The vehicle window region is always segmented into several small zones, it is necessary to combine them together for 
subsequent judgment. As the vehicle windows are located at the upper part of the vehicle image, just the analysis to the upper 
part of the image is required. The procedures for combining the segmented small zones into vehicle windows are as follows: 

1) Among the segmented images, first find the centroid of various small zones, mark various small zones with centroid 
located at upper part of the image, express its set as “region”. 
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2) The width and length of the image is width and height, take the points with coordinate of [width/2, height*3/8] in the 
image as the mass center reference point P of the vehicle window, calculate the Euclidean distance from the mass center 
of region in the set to point P, and sort the regions from small to big based on this distance. If the closer the region 
centroid to the reference point P is, the more likely to belong to the internal window region will be, this can guarantee 
the combination of zones starts from the inside of the vehicle window. 

3) The first small zone in the region is taken as the initial vehicle region, and written as “Re”. Put the small zones in region 
into the Re, and combine them with the zones in Re. After certain zone is merged each time, calculate the area of Re at 
this moment, if the area is larger than the threshold (taking value 9500), this zone will be removed, as it possibly is the 
zone outside the vehicle window; the area is smaller than the threshold, it is considered as a successful combination, 
calculate the boundary smoothness of Re at this moment, save it in the array and save the vehicle window region at this 
moment. Continue to combine the subsequent small zone till the region is processed completely. 


Fig.3 Zone combination 

As shown in Fig.3, the numbers in the image are the marks of zones in the region after sorting, the reference point P of 
centroid for zone 1 is the closest one, then comes zone 2, the rest can be done in the same manner. The zone 1 is combined 
with zones 2 and 3 successively, however the adding of zone 4 will cause the total area to exceed the threshold, therefore the 
zone 4 will not be combined. 

The vehicle window region obtained by using the area to restrain the combination will not only guarantee that the vehicle 
window will not be combined with other part of the vehicle body, but also not miss any zone within the window region, 
ensuring the accuracy of final result. 

4.2 Extraction of candidate vehicle window region 

The shape of vehicle window is approximately trapezoidal, its boundary is consisted of approximately four straight segments, 
and therefore boundary of window region obtained from present algorithm should be also smooth enough. The small zones 
obtained by using the segmentation method in present paper are irregular in shape, when the zones are combined with the 
vehicle window and the boundary of the candidate window region is analyzed, if it is the closer to the actual zones of the 
window, its boundary will me smoother. The present paper calculates the boundary smoothness based on the extraction of 
region boundary chain codes. 

4.2.1 Chain code 

Chain code [14] refers to 8 possible directional values of the line connecting adjacent pixels, for each pixel dot, the adjacent 
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Fig.4 Chain code definition 


dots will use the continuation direction to represent the directional code. 
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The definition of 8 directional chain codes is as shown in Fig.4. If one is added to the chain code value, its direction is that 
rotates by 45° in counterclockwise direction, its direction will be rotate in counterclockwise direction with the increase of 
chain code values. Each pixel has only 8 adjacent dots, the chain code value will be 0 to 7, 8 in all. 

The chain code of region boundary depends upon the start point and the direction of boundary follow-up. The paper takes 
clockwise direction. After binarizing the combined regions, it is easy to obtain the boundary sequence, and find the boundary 
chain code, which is expressed as Ccode. 

4.2.2 Boundary smoothness 

Any point on the boundary has a chain code A1 pointing this point and the chain code A2 pointing to next point. Mutual 
relationship of chain codes A1 and A2 is called as “relative chain code”, the relative chain code will become zero when they 
point to same direction. If A2 is rotated relative to A1 in counterclockwise direction, the relative chain code will take values 
of 1 to 4 according to the size of the angles, corresponding to rotate by 45°, 90°, 135° and 180°; the relative chain code will 
take values of -1 to -3, corresponding to rotate by 45°, 90° and 135° along clockwise direction. 

The paper defines the boundary smoothness tb to measure the degree of boundary that is composed of vehicle window tends 
to be straight line segment. First, calculate the generalized relative chain code Dcode: 

Dcode=Ccode(i+3)-Ccode(i) (6) 

That is to subtract the chain code of current point from last three digits in the chain code. If the boundary is a straight line 
segment at this moment, Dcode is zero. The calculation of tb is as shown in equation (7): 

tb=num(Dcode==0)/num(Dcode) (7) 

That is the ration of number of zero values in relative chain code, num() is the function for number of statistics. 

The larger the tb is, indicates the more smooth the boundary will be, the smaller the degree of bumps on the border, the 
closer the region tends to be a regular shape (trapezoid). Take the region with maximum tb as the candidate window region, 
use 10x10 square structure to conduct morphological open operation, remove the long and further smooth the region 
boundaries as the final candidate window region. 

4.3 Feature judgment 

The region with maximum boundary smoothness may not be the ideal window, therefore it is still necessary to carry out final 
judgment to the candidate window. As the shape of window is approximate to trapezoid, and the difference among the 
windows of different types of cars is not big. The paper judges whether it is the window depending on the area of window 
region area, rectangularity extent and the ratio of length to width t. 

1) By combining the area threshold in case of above-mentioned combination of region, the area of window should satisfy 
5000 < area < 9500. 

2) Calculation of rectangularity 

extent=area/areaBox (8) 

Where, areaBox is the area of smallest circumscribed rectangle of the region. The bigger the rectangularity, the closer the 
region is to the rectangle. 

The window region is approximate to the trapezoid not a rectangle, therefore it will be acceptable is extent >0.85 is satisfied. 

3) The ratio of height to width of car is about 0.5, by taking into consideration that the shooting angle of the image may 
reduce the height of window in the image, the present paper takes the ratio of height to width of 0.25 < t < 0.55. 

If the candidate window region meets above 3 conditions, it is considered that the window is accurately segmented; 
otherwise, it is considered that the extraction of window in current scale of segmentation fails, carry out the analysis of next 
scale till the window is accurately segmented. 

V. Experimental results and analysis 

To verify the effectiveness of the algorithm in present paper, the front image of vehicle taken by the surveillance camera at 
the intersection is used for experiment. There are 720 images covering 20 types of cars with various colors in different 
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lighting environment. The sizes of the images are normalized to 200x220 pixels. The algorithm is realized by MATLAB 
r2013a, and runs under the condition of Intel 3.6G HZ and 2 GB RAM. 

5.1 Experimental evaluation method 

Same as Lu [6] , the present paper evaluates the algorithm in two aspects: locating performance and segmentation performance. 

The locating performance reflects the degree of difference of widow position coordinates obtained by present algorithm from 
the actual coordinates. Here the coordinates of window position refers to the coordinates of circumscribed rectangle in 
window region. The locating error will be measured by the average of distance between corresponding vertices of two 
rectangles, as it is expressed in equation (9): 


Z7 _ (' D lt + D rt + D lb + D rb ) 
^ loc ~ A 


( 9 ) 


Where, E loc represents the locating error, Di t ,D rt ,D lb ,D rb are the distance between corresponding top left, top right, bottom left, 
bottom right vertices of two rectangles respectively. Finally, the locating error has to be normalized by using width of 
image. 

The segmentation performance reflects the degree of fit between the window region and the actual window region obtained 
by this algorithm. This is not only related with accurate expression of the size and the shape of the window, but also related 
with complete extraction of image information in the window region. The segmentation performance is measured by using 
the Jaccard distance between the region obtained by the algorithm and actual region of the window, as it is expressed in 
equation (10): 


seg 


GTnSR 
GT uST? 


( 10 ) 


Where, E seg represents the segmentation error, GT (ground truth) represents actual window region, SR (segmented region) is 
the region from segmentation. 

5.2 Algorithm comparison 

The algorithm in this paper will be compared with the algorithm proposed by Hao [1] , the algorithm of Hao [1] is to find four 
boundaries of the window by combining the edge detection with Hough transform, and locate the extraction of the window. 


window localization window segmentation 



Fig.5 Cumulative locating error Fig.6 Cumulative segmentation error 


In Fig.5, the horizontal axis is the locating error normalized by the width of the image, the vertical axis accounts for the 
proportion in the total number of images. The red line is the performance of the proposed algorithm in this paper, the green 
line is the performance of Hao [1] algorithm. It can be seen that the locating performance of present algorithm is better, 
especially prominent, the proportion of small errors is close to 40%, and this also shows that the present algorithm locating 
effect is good. 
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In Fig. 6, the horizontal axis is the segmentation error, the vertical axis accounts for the proportion in the total number of 
images. The red line is the performance of the present algorithm, the green line is the performance of Hao [1] algorithm. The 
performance of present algorithm is obviously better. This is because this algorithm can only segment partial region of the 
vehicle window, however the segmentation results of the present algorithm can be better fit the edge of the window region. 


5.3 Effect of algorithm 

Some segmentation effects of the algorithm in this paper are shown in Fig. 7, the abstraction of window is marked with a blue 
area. It can be seen that this algorithm has a good segmentation effect for various colors of vehicles. In lower half part of 
Fig.7, two groups at left and right are original images of vehicles and their window segmentation images under darker and 
stronger light conditions respectively. It can be known from the image that this algorithm can also have much better locating 
and segmentation effect even under the influence of different light, and shows better robustness for the test photos. 


It can be known from Fig.7 that this algorithm can not only locate the position of window accurately, but also much better fit 
the segmented vehicle window region with edges of original vehicle window, and maintain the shape and size of the original 
window. This feature allows the window obtained by this algorithm can be applied in analysis of object in the window, such 
as detection of safety belt and number of passengers, but also further extract the window shape feature to apply for vehicle 
type recognition 



Fig.7 Original image of vehicle and window segmentation image 


VI. Conclusion 

This paper has analyzed the advantages and disadvantages of current vehicle window location algorithm, proposed a vehicle 
window segmentation algorithm based on region segmentation and boundary constraint. First the region segmentation shall 
be carried out for the car image according to the color feature, then the small regions will be combined and merged, finally 
the window region is obtained by combining the geometrical features of the vehicle window. The algorithm result not only 
locates the vehicle window accurately, but also keeps the shape and size of the window, which has laid a foundation for 
subsequent application of vehicle window image. The experiments show that the algorithm has better segmentation results 
for cars with different colors and different lighting conditions. 
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